1. Field of the Invention
This invention is related to the field of computer systems and, more particularly, to computer systems wherein input/output (I/O) operations access memory.
2. Description of the Related Art
Generally, personal computers (PCs) and other types of computer systems have been designed around a shared bus system for accessing memory. One or more processors and one or more input/output (I/O) devices are coupled to memory through the shared bus. The I/O devices may be coupled to the shared bus through an I/O bridge which manages the transfer of information between the shared bus and the I/O devices, while processors are typically coupled directly to the shared bus or are coupled through a cache hierarchy to the shared bus.
Unfortunately, shared bus systems suffer from several drawbacks. For example, the multiple devices attached to the shared bus present a relatively large electrical capacitance to devices driving signals on the bus. In addition, the multiple attach points on the shared bus produce signal reflections at high signal frequencies which reduce signal integrity. As a result, signal frequencies on the bus are generally kept relatively low in order to maintain signal integrity at an acceptable level. The relatively low signal frequencies reduce signal bandwidth, limiting the performance of devices attached to the bus.
Lack of scalability to larger numbers of devices is another disadvantage of shared bus systems. The available bandwidth of a shared bus is substantially fixed (and may decrease if adding additional devices causes a reduction in signal frequencies upon the bus). Once the bandwidth requirements of the devices attached to the bus (either directly or indirectly) exceeds the available bandwidth of the bus, devices will frequently be stalled when attempting access to the bus, and overall performance of the computer system including the shared bus will most likely be reduced.
On the other hand, distributed memory systems lack many of the above disadvantages. A computer system with a distributed memory system includes multiple nodes, two or more of which are coupled to different memories. The nodes are coupled to one another using any suitable interconnect. For example, each node may be coupled to each other node using dedicated lines. Alternatively, each node may connect to a fixed number of other nodes, and transactions may be routed from a first node to a second node to which the first node is not directly connected via one or more intermediate nodes. A memory address space of the computer system is assigned across the memories in each node.
In general, a xe2x80x9cnodexe2x80x9d is a device which is capable of participating in transactions upon the interconnect. For example, the interconnect may be packet based, and the node may be configured to receive and transmit packets. Generally speaking, a xe2x80x9cpacketxe2x80x9d is a communication between two nodes: an initiating or xe2x80x9csourcexe2x80x9d node which transmits the packet and a destination or xe2x80x9ctargetxe2x80x9d node which receives the packet. When a packet reaches the target node, the target node accepts the information conveyed by the packet and processes the information internally. Alternatively, a node located on a communication path between the source and target nodes may relay the packet from the source node to the target node.
Distributed memory systems present design challenges which differ from the challenges in shared bus systems. For example, shared bus systems regulate the initiation of transactions through bus arbitration. Accordingly, a fair arbitration algorithm allows each bus participant the opportunity to initiate transactions. The order of transactions on the bus may represent the order that transactions are performed (e.g. for coherency purposes). On the other hand, in distributed systems, nodes may initiate transactions concurrently and use the interconnect to transmit the transactions to other nodes. These transactions may have logical conflicts between them (e.g. coherency conflicts for transactions involving the same address) and may experience resource conflicts (e.g. buffer space may not be available in various nodes) since no central mechanism for regulating the initiation of transactions is provided. Accordingly, it is more difficult to ensure that information continues to propagate among the nodes smoothly and that deadlock situations (in which no transactions are completed due to conflicts between the transactions) are avoided.
In a computer system having a distributed memory system, memory operations (e.g., read and write operations) originating from I/O nodes may need to be properly ordered with respect to other pending memory operations in order to preserve memory coherency within the computer system and to satisfy ordering requirements expected by the I/O nodes. For example, memory operations may need to be completed in the order in which they were generated (i.e., in chronological order) in order to preserve memory coherency within the computer system and to satisfy I/O ordering requirements. It would thus be desirable to have a computer system implementing a system and method for properly ordering I/O memory operations.
A computer system is presented which implements a system and method for ordering input/output (I/O) memory operations. In one embodiment, the computer system includes a processing subsystem and an I/O subsystem. The processing subsystem includes multiple processing nodes interconnected via coherent communication links. Each processing node may include a processor preferably executing software instructions (e.g., a processor core configured to execute instructions of a predefined instruction set). The I/O subsystem includes one or more I/O nodes coupled via non-coherent communication links one after another in series or daisy chain fashion. Each I/O node may embody one or more I/O functions (e.g., modem, sound card, etc.). One of the processing nodes includes a host bridge which translates packets moving between the processing subsystem and the I/O subsystem. One of the I/O nodes is coupled to the processing node including the host bridge via a non-coherent communication link.
The I/O node coupled to the processing node produces and/or provides transactions having a destination or target within the processing subsystem to the processing node including the host bridge. The I/O node may, for example, produce and/or provide a first transaction followed by a second transaction. The host bridge may dispatch the second transaction with respect to the first transaction according to a predetermined set of ordering rules. For example, the host bridge may: (i) receive the first and second transactions from the I/O node, (ii) dispatch the first transaction within the processing subsystem, and (iii) dispatch the second transaction within the processing subsystem dependent upon progress of the first transaction within the processing subsystem according to the predetermined set of ordering rules.
The computer system may include one or more memory units within the processing subsystem. For example, the computer system may include multiple memories such that each memory is coupled to a different one of the multiple processing nodes. The first transaction may be a write transaction for storing data within a first memory coupled to a first processing node. In response to the first transaction, the first processing node may produce a target start packet directed to the host bridge (e.g., after ordering the first transaction with respect to other transactions pending within the first processing node). The second transaction may be a write transaction for storing, data within a second memory coupled to a second processing node. Acting in accordance with the predetermined set of ordering rules, the host bridge may delay dispatching the second transaction until the target start packet resulting from the first transaction is received.
In a second embodiment of the computer system, the host bridge may delay completion of the second transaction with respect to the first transaction according to the predetermined set of ordering rules. For example, the first transaction may be a posted write transaction for storing data within the first memory. In response to the first transaction, the first processing node may produce a first target done packet directed to the host bridge. For example, the first processing node may produce the first target done packet after the first transaction has reached a xe2x80x9cpoint of coherencyxe2x80x9d within the computer system. The first transaction may have reached the point of coherency within the computer system when the first transaction is: (i) properly ordered within the first processing node with respect to other transactions pending within the first processing node, and (ii) a correct coherency state with respect to the first transaction has been established in the other processing nodes.
The second transaction may be a Flush command from the same I/O node which issued the first transaction. The Flush command may ensure that posted write transactions issued prior to the Flush command by the I/O node have at least reached points of coherency within the computer system. The second transaction may be complete when the host bridge dispatches a second target done packet to the I/O node. Acting in accordance with the predetermined set of ordering rules, the host bridge may delay issuance of the second target done packet to the I/O node until the first target done packet resulting from the first transaction is received from the first processing node.
In a first method for use in the computer system described above, the I/O subsystem provides a first transaction and a second transaction to the host bridge, wherein the second transaction follows the first. The host bridge: (i) dispatches the first transaction within the processing subsystem, and (ii) dispatches the second transaction within the processing subsystem dependent upon progress of the first transaction within the processing subsystem and the predetermined set of ordering rules. In a second method for use in the computer system described above, the I/O subsystem provides the first and second transactions to the host bridge. The host bridge: (i) dispatches the first transaction within the processing subsystem, and (ii) delays completion of the second transaction with respect to the first transaction according to the predetermined set of ordering rules.